System and method of packaging computing resources for space and fire-resistance

ABSTRACT

A computing system includes one or more rack rows comprising one or more racks. The racks includes one or more tanks that hold liquid coolant for at least one of the one or more servers, and a liquid coolant to remove heat from at least one of the one or more servers. An aisle is provided next to a rack row or between two of the rack rows. The aisle includes a floor. The floor can be walked on by service personnel to access at least one of the one or more racks in at least one of the rows. Cooling components at least partially below the aisle move a liquid to remove heat from at least one of the servers in at least one of the racks. The racks, floor and cooling components may be fire-resistant.

PRIORITY CLAIM

This application claims the benefit of U.S. Provisional Application Ser.No. 61/854,949 entitled “Methods for packing a carnotJet pump module forspace and or fire resistance” filed May 6, 2013, which is incorporatedherein by reference in its entirety.

BACKGROUND

1. Field

The present invention relates generally to providing resources forcomputing. More particularly, the present disclosure relates to systemsand methods for packaging computing and associated components for spaceefficiency and/or fire resistance.

2. Description of the Related Art

A data center typically includes a group of computing devices at acommon physical location. Data centers are often housed in conventionalbuilding structures and use air conditioning systems to remove heatgenerated by electronic components (chips, hard drives, cards, etc.)

Many commercially-available servers used in data centers are designedfor air cooling. Such servers usually comprise one or more printedcircuit boards having a plurality of electrically coupled devicesmounted thereto. These printed circuit boards are commonly housed in anenclosure having vents that allow external air to flow into theenclosure, as well as out of the enclosure after being routed throughthe enclosure for cooling purposes. In many instances, one or more fansare located within the enclosure to facilitate this airflow.

Data centers housing such servers and racks of servers typicallydistribute air among the servers using a centralized fan (or blower). Asmore fully described below, air within the data center usually passesthrough a heat exchanger for cooling the air (e.g., an evaporator of avapor-compression cycle refrigeration cooling system (or “vapor-cycle”refrigeration), or a chilled water coil) before entering a server. Insome data centers, the heat exchanger has been mounted to the rack toprovide “rack-level” cooling of air before the air enters a server. Inother data centers, the air is cooled before entering the data center.

In general, electronic components of higher performing servers dissipatecorrespondingly more power. However, power dissipation for each of thevarious hardware components (e.g., chips, hard drives, cards) within aserver can be constrained by the power being dissipated by adjacentheating generating components, the airflow speed and airflow paththrough the server and the packaging of each respective component, aswell as a maximum allowable operating temperature of the respectivecomponent and a temperature of the cooling air entering the server asfrom a data center housing the server. The temperature of an air streamentering the server from the data center, in turn, can be influenced bythe power dissipation and proximity of adjacent servers, the airflowspeed and the airflow path through a region surrounding the server, aswell as the temperature of the air entering the data center (or,conversely, the rate at which heat is being extracted from the airwithin the data center).

It requires a substantial amount of space to house data centers inconventional buildings. In addition, servers deployed in buildings maynot portable and may be expensive, as energy costs and power dissipationcontinue to increase. Air cooling of a data center is also spaceintensive, because the efficiency of cooling is affected by theproximity of electronic components.

Electrical or other ignition sources sometimes cause fires in datacenters. Data center fires can be quite costly events, since a fire caneasily and quickly spread to damage numerous pieces of expensiveequipment packed into a small area. The oils used to cool servers willcatch fire under certain conditions. In some cases, a fire can spreadnot only in oil in tanks holding rack-mounted servers, but also to oilspilled in around the racks during installation or maintenance.

SUMMARY

Embodiments of systems and methods of packaging and operating computingresources are described herein. In an embodiment, a computing systemincludes one or more rack rows comprising one or more racks. The racksincludes one or more tanks that hold liquid coolant for at least one ofthe one or more servers, and a liquid coolant to remove heat from atleast one of the one or more servers. An aisle may be provided next to arack row or between two of the rack rows. The aisle may include a floor.The floor can be walked on by service personnel to access at least oneof the one or more racks in at least one of the rows. Cooling componentsat least partially below the aisle move a liquid to remove heat from atleast one of the servers in at least one of the racks.

In an embodiment, a method of packaging and providing an operatingenvironment for computing resources includes: providing one or more rackrows including one or more racks, wherein at least one of the racksincludes one or more servers; at least partially filling at least someof the racks with liquid coolant; and providing one or more coolingcomponents at least partially below the racks, wherein at least one ofthe cooling components moves a liquid to remove heat from at least oneof the servers in at least one of the racks.

In an embodiment, a cooling module includes one or more pumps and anenclosure. The pumps move at least one liquid to remove heat from liquidcoolant in at least one rack. The top of the enclosure iscoolant-permeable such that liquid coolant can pass through the top intothe enclosure. The cooling module top is configured to serve as a floorfor service personnel standing on the top of the enclosure to accessservers in the racks.

In an embodiment, a data center includes one or more racks, a floor, anda cooling component. The cooling component is under the floor. Thecooling component moves a liquid to remove heat from at least one of theservers in at least one of the racks. The floor supports servicepersonnel while the service personnel access racks. The floor includes acoolant-permeable and fire-suppressing material, such as perviousconcrete.

In an embodiment, a computing module includes one or more racksincluding servers and one or more tanks configured to hold liquidcoolant for the servers. Liquid coolant held the tanks removes heat fromthe servers. A cooling component moves the liquid coolant in the tankssuch that a least a portion of the liquid circulates within the tank. Aportion of the circulating fluid passes through or across at least oneof the servers to remove heat from the at least one of the servers. Theliquid coolant circulates such that substantially all of the liquidcoolant on the surface of the liquid coolant contained in the at leastone tank is in motion when the cooling component is operating.

In an embodiment, a computing system includes one or more rack rowscomprising two or more racks. A cable tray spans across two or more ofthe racks in at least one of the rack rows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an embodiment of computing system including enclosedcooling module for maintaining racks under the floor of an aisle betweenthe racks.

FIG. 1B illustrates one embodiment of an exemplary system forefficiently cooling a plurality of independently operable dataprocessing modules.

FIG. 1C illustrates an alternative embodiment of an exemplary system forefficiently cooling a plurality of independently operable dataprocessing modules.

FIG. 1D illustrates a schematic plan view of one embodiment of a datacenter including cable trays extending across rows of racks.

FIG. 2 illustrates a rack that can be used in a computing module invarious embodiments.

FIG. 3 illustrates one embodiment of a computing module including rackswith liquid-cooled servers, power distribution components, and anunder-floor cooling module.

FIG. 4 is a schematic view illustrating circulation at a section of arack in one embodiment.

FIG. 5A is a schematic end view illustrating an embodiment of rack in adata center with a pump module in between rack rows.

FIG. 5B is a schematic plan view illustrating an embodiment of rack in adata center with a pump module in between rack rows.

While the invention is described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the invention is not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit the invention tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention as defined by the appendedclaims. The headings used herein are for organizational purposes onlyand are not meant to be used to limit the scope of the description orthe claims. As used throughout this application, the word “may” is usedin a permissive sense (i.e., meaning having the potential to), ratherthan the mandatory sense (i.e., meaning must). Similarly, the words“include”, “including”, and “includes” mean including, but not limitedto.

DETAILED DESCRIPTION OF EMBODIMENTS

In some embodiments, a method for integrating a computing systemincludes positioning a pump module for a set of liquid cooled racks suchthat the pump module fits under a floor. The racks and the pump modulemay include provisions to reduce the flammability of the unit. In oneembodiment, the pipes in between the rack and the pump module are madeof steel or another temperature-resistant material such that if a firebreaks out, the oil carrying pipes will not melt or catch fire, and willcontain the fluid. The outside of the pump module may be made of steel.In some embodiments, a cooling module circulates liquid coolant in racksin a manner that inhibits and suppresses fire in the racks.

In some embodiments, an enclosure for a pump module includes gaskets toseal the components of the pump module in a closed volume. Sealing apump module may suppress fire in the pump module (for example, limitingthe supply of air in the enclosure.)

In various embodiments, a plurality of servers is mounted vertically ina rack. The servers may be vertically removed and replaced from a rackwith an open top. The servers may be mounted in an array, arrangedhorizontally. Each server may be removed without affecting thefunctionality of other servers in the rack or in the data center. Eachserver may operate independently of each of the servers.

In some embodiments, spacing and structure allow for accommodation ofmany different form factors, including but not limited to conventionalrack mount servers normally used for air cooling. Servers may be mountedadjacently to each other to minimize upward flow around the servers ormotherboards.

In some embodiments, dielectric fluid is pumped out of a rack, cooled byflowing through a heat exchanger, and pumped back into the rack. Inother embodiments, the heat exchanger is located inside the rack. Asecondary liquid circuit flows into the rack with dielectric fluid, andthrough a heat exchanger, cooling the dielectric fluid.

In one embodiment, a dielectric coolant flows out of a rack at anelevated temperature. A circuit may include a pump, heat exchanger, andmeasurement devices. A secondary circuit flows through the above heatexchanger, cooling the dielectric fluid, then flows outside anddissipates heat external to the room housing the dielectric-filled rack.

Systems and methods of cooling and operating electronic devices usingliquid coolant-filled racks may be as described in US Patent PublicationNo. 2011/0132579 (the “'579 Publication”), by Best et al., publishedJun. 9, 2011, which is incorporated by reference in its entirety as iffully set forth herein.

Computing System with Under-Floor Cooling Module and Fire Resistance

In some embodiments, a computing system includes a row of liquid-cooledracks on either side of an aisle. A cooling module is included under afloor of the aisle.

FIG. 1A illustrates one embodiment of a computing system with anunder-floor cooling module between two rows of racks. Computing system110 includes racks 112. A cooling system 114 is provided for each rack.Each of racks 112 is supplied power through a power distribution unit116. Cable trays 118 hold cables that supply power or enable exchange ofdata between servers in racks 112 and external systems.

Each of racks 112 includes tank 122. In the embodiment shown in FIG. 1A,a plurality of tanks 122 are provided, each tank 122 containingvertically mounted, independently removable and replaceable dataprocessing modules. As shown in FIG. 1A, in this embodiment, tanks 122are arranged in two banks adjacent an aisle 124. The tanks may bearranged in other configurations, however. For example, a single bank oftanks 122 may be installed in the center of the unit with aisles oneither side of tanks 122. Or a single bank of tanks 122 may be installedagainst a wall of an enclosure (e.g., a shipping container housing racksand the cooling module.)

Cooling system 114 includes pumps 130, one or more heat exchangers, andassociated pipes and control systems. Pumps 130 and the heatexchanger(s) are included in the form of pump module 135. Pump module135 includes enclosure 136. Enclosure 136 houses the pumps and othercomponents of pump module 135.

The pipes in between the rack and the pump module may be made of steelor another temperature resistance material such that if a fire breaksout, the oil carrying pipes will not melt and will contain the fluid. Inaddition, the enclosure for the pump module and the rack housing may bemade of a fire-retardant material, such as steel.

In some embodiments, the floor of aisle 124 (which, in the embodimentshown in FIG. 1, is the top of the enclosure for the cooling module) ispermeable such that coolant (for example, oil) can drip through thefloor into the cooling module. Allowing coolant on the floor to seepinto a porous material may extinguish or suppress fire (for example, byisolating the oil in the pores of the material from air).

The top of a pump module and any removable service panels for a pumpmodule may be made of flame resistant material. The top of the pumpmodule may have a permeable surface such that any coolant (for example,oil) that has spilled into the aisle drains drips through the top andpasses into pump module. In certain embodiments, the system includes oneor more trays or pans that collect spilled fluid around the rack or pumpmodule and drain it into the pump module.

As shown in FIGS. 1A and 1B, module 135, which may include at leastelements as shown in FIG. 1B, for example, provided cooling for tanks122, according to one or more embodiments. That is, according to one ormore embodiments, each pump module 135 may include primary and secondarypumps 130 (and associated pump motors) connected to filter 160 andliquid coolant heat exchanger 140 of at least one bank of tanks 122 viafluid circuit 170 such that primary and secondary pumps 130 may functionindependently of one another for backup purposes, with electricallyisolated pump 130 motors. According to one or more embodiments, primarypump 130 motor is controlled by variable speed controller 180 forregulating temperature of coolant loop 170 by varying liquid coolantflow, whereas secondary pump motors may be fixed-speed and controlled byon-off control.

Module 135 for evaporative cooling apparatus 150, according to one ormore embodiments, includes a controller for controlling a pump motor inloop 175, which may be on-off control or variable speed control,according to one or more embodiments, and includes a controller for oneor more fans, motor, which may be like controller 180 of FIG. 1B, forexample, but for regulating fans speed of evaporative cooling apparatus150 in order to control temperature of cooling loop 175 by varying airflow over evaporative final heat exchanger 152. A pump, motor controllerand cooling water loop may also be provided to run water over theexterior of a heat exchanger external to module 135 for additionalcooling.

Controllers 180 may be interfaced via a network with a master controllerfor which a single dashboard is provided, according to one or moreembodiments, which is for displaying and controlling water flow in oneor more loops through one or more cooling towers, fan power for air flowacross the one or more heat exchangers, one or more cooling towers, andliquid coolant flow in one or more loops for tanks 122. Preferably, amaster controller optimizes all elements for minimum power consumptionof the system while maintaining sufficient cooling. The networkcontroller performs diagnostic testing of each element separately forfunctionality and reports the functionality back to a single user. Thissingle management point makes the system more reliable and moreefficient, since the master controller can obtain maximum efficiency forall components. In some embodiments, control is carried out as describedin the '579 Publication.

A control module may connect into a larger monitoring system, such as abuilding management system, data center management system, stand aloneor other manner of operations. In some embodiments, the control modulecontrols and monitors external components, such as building water pumps,cooling towers, remote battery backup components, other modules, remotepower generators, security, room PDUs, rack PDUs, and other systems.

Power distribution units 116 mount near the back of the racks 112, underthe lid area. In this embodiment, the only in/out cable in the rack maybe the main power feed(s) for the PDU. This arrangement may minimize theamount of space necessary for users to service the back of the rack.

A single or multiple power feed(s) may feed electronic systems in themodule. The power feed may be at voltages such as 208, 240, 277, 480VAC, or DC voltages in the case of systems using DC battery backup ordistribution system. A single feed may go into a power distributioncenter, or multiple power distribution centers or subpanels, to connectall required loads and/or if redundancy is required.

The power distribution system may include a transformer that adjusts(and, if required, isolates) the input AC voltage to the required loadvoltages. The power may be distributed into multiple ports. Each or agroup may include a breaker. Each port may connect, for example, to arack-level power rack distribution device (e.g. outlet strip) attachedto the racks and/or the liquid-filled rack system. Alternatively, loadsmay be wired into the distribution system directly or other manner ofconnection. In some embodiments, power distribution components mayadjust DC power from a distribution voltage to a load voltage.

Referring now to FIG. 1B, computing system 110 may include a coolingsystem 185 for transferring heat from data processing modules 310. Theliquid coolant heated by data processing modules 310 is fluidly coupledthrough suitable piping or lines to a pump 130, which pumps the heatedliquid coolant through suitable piping or lines to a heat exchanger 140associated with a heat-rejection or cooling apparatus 150. In someembodiments, heat exchanger 140 is remotely or distally located fromtank 122 and/or computing system 110. Heat exchanger 140 rejects theheat from the incoming heated liquid coolant and fluidly couples thecooled liquid coolant through a return fluid line or piping 170 backinto the tank 122. Thus, at least a portion of the liquid coolantcompletes a fluid circuit through the data processing modules 310 intank 122, pump 130, heat exchanger 140, and back into tank 122. The heatrejected from the heated liquid coolant through the heat exchanger 140may then be selectively used by alternative heat rejection or coolingapparatus 150 to dissipate, recover, or beneficially use the rejectedheat depending on the different environmental conditions or dataprocessing modules 310 operating conditions to which the system issubject.

Referring now to FIG. 1C, an embodiment of an alternative cooling system195 is illustrated for cooling data processing modules 310. Unlike thecooling system 185, heated liquid coolant does not flow outside the tank122. Instead, one fluid circuit 260 of the flowing liquid coolant iscompletely internal to the tank 122. A thermal coupling device 280, suchas a heat exchanger, is mounted within the tank 122 within the fluidcircuit through the data processing modules 310, so that at least aportion of the heated liquid coolant flow exiting the data processingmodules flows through the thermal coupling device 280. Cooled liquidcoolant exits the coupling device 280 and at least a portion of thecooled dielectric coolant circulates in the internal fluid circuit 260back through the data processing modules 310.

Cooling systems 185 (FIG. 1B) and 195 (FIG. 1C) include a computercontroller 180 with suitable applications software for implementingvarious embodiments. A detailed description of controller 180 isincluded in international published patent application WO 2010019517which is incorporated by reference as if fully set forth herein. In someembodiments, temperatures of operation may be established and maintainedas set forth in the WO 2010019517 application.

Referring now to FIG. 1B, cooling apparatus 150, which may provides anevaporative final heat exchanger and a motor driven fan for forcing airflow through the final heat exchanger, is located sufficiently far awayfrom tanks 122 to enable adequate heat dissipation at the heat exchangerto cool the heated liquid in loop 175. The resulting heat may be ventedto the ambient outside environment. Alternately, the resulting heat maybe beneficially used, as described in PCT patent application WO2013022805. The cooled liquid is then recirculated through the returnpipe in loop 175 to cool the liquid coolant in loop 170 which, in turn,cools the data processing modules 310 in tanks 122. In some embodiments,cooling apparatus 150 is mounted on the exterior top of a container forthe computing system.

Although one cooling apparatus 150 is shown, more than one may beprovided in various embodiments. For example, one cooling apparatus 150may be provided for each bank of tanks 122. Further, cooling loops (forexample, cooling loop 175) may be arranged, and each cooling apparatus150 may be sized, so that a plurality of cooling apparatus 150 mayprovide backup cooling for one other. Cooling apparatus 150 need not beattached to the shipping container.

In some embodiments, two or more racks are mounted next to each other,and cable trays extend across two or more racks. The individual racksand cable tray may combine to form one rack. For example, as shown inFIG. 1A, cable tray 118 may extend across two of racks 112.

In certain embodiments, cable trays hold cables that run from one end ofthe data center to the other (as opposed, for example, to the cabletrays in the rack that hold cables that run between components in therack (server to rack mount switch). FIG. 1D illustrates one embodimentof a data center with cable trays extending across a data center. Cabletrays 118 in data center 221 span across racks 112 to a common wallincluding power distribution system 223, which provides power to each ofthe rows of racks via cables in cable trays 118.

In one embodiment, a single high voltage power feed goes into one ormore power distribution units (PDU). A transformer reduces the powerfrom high voltage to a lower voltage (for example, 480 VAC to 208 VAC inthe US). The power distribution may include protection against powerspike and/or line noise, such as transient voltage surge suppression.The reduced voltage lines (e.g., 208 VAC) may be distributed intomultiple different feeds.

Each feed may be connected to a cabinet distribution unit (CDU) at therack. Each feed may include a breaker and/or current monitoring. Incertain embodiments, one or more of these components are separated out(transformer, transient voltage surge suppression (“TVSS”), and others),rather than housed inside of a power distribution unit.

In some embodiments, CDUs are included and mounted to each rack. TheCDUs may be passive, active, or combination thereof. An active CDU mayinclude power level measurement of voltage and current for each plug(normally where the server plugs into) and the capability to turn eachport on or off. Monitoring may be performed by control system for themodule. Monitoring may include values for CDU, PDU, or both.

FIG. 2 illustrates a rack that can be used in a computing module invarious embodiments. Rack 222 includes power distribution units 223,liquid coolant inlet port 224, liquid coolant exit port 225, and rackcontrol module 226.

Rack 222 includes a cable management system. Cable management systemincludes cable management rails 227. Each of cable management rails 227includes a series of cable guides. The cable guides may be used to guideand support power cables or data cables for each of the servers in rack232. In some embodiments, cables and cable connector receptacles remainabove the surface level of the coolant in the rack.

FIG. 3 illustrates one embodiment of a computing module including rackswith liquid-cooled servers, power distribution components, and anunder-floor cooling module. In some embodiments, liquid-cooled servers,power distribution components, and liquid cooling components may shiptogether as an assembly. Power distribution components and liquidcooling components may be sized for the servers to be operated in thecomputing module.

Computing module 230 includes racks 232, cooling component 239, andpower distribution components (power distribution components may beunder the floor of the aisle). Each of racks 232 includes servers 240.Servers 240 are vertically arranged in racks 232. In some embodiments,any of servers 240 may be removed from rack 232 without removing ordisturbing operation of the other servers when the top is removed orfolded back. Power distribution units may feed power to cabinet powerdistribution units on each of racks 232.

A computing module of several liquid submersion cooling racks and onepump module (such as computing module 230) may maximize efficiency anddata center floor space. Each block of four racks may have its owncontrol system that optimizes coolant flow in real-time for the givenheat load while monitoring the cooling system across more thantwenty-five parameters.

In some embodiments, OEM and ODM servers are installed vertically intorack 232. Racks 232 may support servers from any manufacturer, in, forexample, Standard 19″ or Open Compute Standard form factors. Servers maybe lowered vertically into the liquid-filled rack. PDUs may be mountedto the front or the back of the rack.

Cooling component 239 may include coolant pumps, filters, andcoolant-to-water heat exchangers. The pump module is responsible forcirculating the liquid coolant and drawing heated coolant through heatexchangers to remove server heat from the racks. The pump module maythen filter the coolant and return the reduced temperature coolant tothe liquid submersion cooling rack. The pump module may establish astable and uniform cooling environment for servers 240 that iscontrolled to ±1° C. throughout each of racks 232.

Cooling component 239 may receive power connectivity and waterconnectivity from the facility. Cooling component 239 may be configuredto use virtually any form of water available in a data center facility.In some embodiments, a pump module includes an independent secondarysystem for backup. If the primary pump should fail, the secondary kickson instantaneously and cooling will continue undisturbed. Although pumpmodule 239 is shown for illustrative purposes at the end of rows ofracks 232, depending on the space requirements of the facility, the pumpmodule may be installed adjacent to the racks, under the floor, or inthe data center periphery.

Coolant may flow in and out of racks 232 at the ends of racks 232(coolant lines for the rearmost pair of racks may also be coupled tocooling component 239, however, they are omitted from FIG. 3 forclarity.) In one embodiment, the overall path of coolant flow is asshown in FIG. 3. Coolant coming into rack 232 may be distributed acrossthe length of rack 232 in high pressure manifold 245 and then channeleddownward (for example, via ducts, barriers, or baffles) to the bottom ofrack 232. In some embodiments, high pressure manifold 245 includes aseries of nozzles pointing downwardly in the rack. The nozzles may causecirculation through or across each of servers 240.

Nozzles may be spaced across the length of manifolds 245 and 247, suchthat a similar circulation is achieved across the length of the manifold(for example, the flow at each server position may be similar). Incertain embodiments, a nozzle is provided corresponding to each serverlocation. Nozzles may nevertheless, be spaced at any interval, regularor irregular, to achieve desired flow characteristics (e.g., closertogether or farther apart than one nozzle per rack position).

From the bottom of rack 232, coolant may flow up between or throughservers 240 until it reaches the top of servers 240 (and near thesurface of liquid coolant), increasing in temperature as heat istransferred from heat producing components on the servers. Some of thecoolant may be drawn back toward the high-pressure manifold side of therack. A portion of the coolant near the surface may be drawn out of rack232 through suction manifold 247 and returned to a heat exchanger incooling component 239.

Downward flow exiting the nozzles under the manifold may draw liquid atthe surface of the bath downward (for example, in the direction ofarrows shown in FIG. 4). In some embodiments, suction at the surface ofthe liquid resulting from the high pressure nozzles is such that thelevel of liquid on the high pressure manifold side is lower than thelevel of the liquid on the suction side.

In various embodiments, augmentation devices (such as nozzles, pumps, orfans) are provided at multiple locations in a rack. For example, incertain embodiments, nozzles or pumps may be provided at the bottom ofeach of servers 232.

In some embodiments, a controller provides diagnostics and controls forcomputing module 230. The control system optimizes coolant flow toprovide the most efficient coolant flow at all times for the given heatload. The control module may also initialize the backup system andprovide alerts in the event of system downtown or failure. In variousembodiments, the controller may provide temperature analysis, pressureand coolant level verification, power consumption, smart monitoring, anddiagnostics. Controller outputs may include log files of the aboveparameters, e-mail and SMNP diagnostic alerts, and hourly statuscondition updates. This information may also be made available via anetwork as well as a secure internet portal.

Coolant Flow Pattern for Fire-Resistance

In some embodiments, a flow pattern in the rack is such that the fluidflow goes from one side of the top of the rack to other (horizontally).This flow may be formed by fluid rising through the server, coming outof the server, travelling horizontally, and then traveling back downwardoutside of the server. If the top of the rack catches fire, cooled oilconstantly rises to the top. This fluid motion may ensure the hot fluidand cold fluid does not separate in the event of a fire.

Because the oil may have a flash point (150 C) much higher thanoperating temperature of the rack (for example, 40 C), the cold fluid(40 C) may cool the fluid on fire (>150 C). The cooling may reduceenergy of the fire or reduce the surface temperature of the oil belowthe flash point, extinguishing the fire.

FIG. 4 is a schematic view illustrating circulation at a section of arack in one embodiment. The pattern in FIG. 4 may be the same at eachserver position in the rack. Liquid coolant is introduced under pressurethrough nozzle 251 on the underside of high-pressure manifold 245. Highpressure manifold 245 causes circulation of part of the coolant in agenerally down-across-up-across pattern within the bath of liquidcoolant. The arrows in FIG. 4 illustrate the general pattern of theflow.

Coolant from nozzle 251 may combine with coolant being drawn from thesurface of the coolant and moved near the bottom of the tank, below thebottom of servers 240. Coolant in the bottom of the tank may be forcedupwardly through or across servers 240, removing heat fromheat-producing components (for example, processors, semiconductordevices) of the servers. Similar to the manner described relative toFIG. 3, near the surface of the coolant, some of the coolant may bedrawn back toward the high-pressure manifold side of the rack. A portionof the coolant near the surface may be drawn out of rack 232 throughsuction manifold 247 and returned to a heat exchanger in coolingcomponent 239.

Suction manifold is located against the wall of rack 232. All of thecoolant at the surface is thus in a contiguous body between suctionmanifold 247 and the opposing wall of the rack. In some embodiments, thecooling system moves the liquid coolant such that substantially all ofthe liquid coolant at the surface is in motion (for example, eitherright toward the wall on the high-pressure manifold side or left towardthe wall on the suction manifold side.) In this manner, the amount ofstationary liquid at the surface of the bath may be eliminated orminimized. Motion of the liquid (rather than, for example, stationaryfluid) may inhibit fire at the surface of the coolant (e.g. by reducingor minimizing hot spots and/or reducing temperatures below the flashpoint of the oil). In some embodiments, operation of the cooling systemto move the liquid coolant inhibits dead zones at the surface of thecoolant in the rack.

Cover 253 of rack 232 may be made of a fire-retardant material, such assteel. In addition, cover 253 may be non-porous such that the lidinhibits air from entering into the interior of the rack from theoutside. In some embodiments, the cover is sealed (for example, by wayof a gasket between the sides of the rack and the cover. When cover 253is closed, cover 253 may suffocate any fire that starts in the interiorof rack 232.

FIG. 5A is a schematic end view illustrating an embodiment of rack in adata center with a pump module in between rack rows. FIG. 5B is aschematic plan view illustrating an embodiment of rack in a data centerwith a pump module in between rack rows. The top of the pump module isdesigned to be a walking surface with surface tiles that can be removedto service equipment. (Tiles may be for example, may be as shown as inFIG. 3, in which tiles 255 form floor 257). Surface tiles may have anon-slip surface. Pieces of the top of a pump module(s) (for example,tiles such as tiles 255) can be removed to enable service personnel toaccess the pumps and/or equipment in the pump module that must beserviced. In addition, the top of the pump module may serve as a floorfor personnel to access the racks. Stands may position racks at a heightrelative to floor to enable service personnel standing on the floor topull servers out of the rack.

Computing system 300 includes racks 302, pump module 304, and electricaldistribution module 306. Each of racks 302 is mounted on one of stands308. The height that the racks are placed at may be selected to enableconvenient access of the racks by service personnel standing on the topsurface of pump module 306. Racks 302, pump module 304, and electricaldistribution module 306 may be similar to those described above relativeto FIGS. 1A through 4. Each of racks 302 includes power distributionunits (“PDUs”) 310. Water pipes 314 and cables 316 may be run under oneor more racks 302 via the space provided by stands 308. In someembodiments, stands for supporting a rack are integrated into one ormore racks, a pump module, or both.

In the enclosed cooling module, some heating occurs due to the heatproduced by components operating in the module, or from the heatedcoolant. For example, pump/motor systems in a pump module enclosure mayproduce heat due to inefficiency. The motors may be air cooled and dumpheat into the air, which is mostly sealed inside of the pump module. Incertain embodiments, a radiator is provided in the enclosure (forexample, attached inside of the pump module) with oil or water (thesecondary circuit) flowing through the radiator. The radiator transmitsheat from ambient air inside the pump module into the oil. The heatgenerated by the motors is cooled by the radiator.

In some embodiments, a fluid level sensor is provided that monitors ifthe liquid level gets too high. For example, if coolant accumulates in apump module enclosure above a pre-determined level, a controller mayautomatically shut off the pump or other systems, sound an alarm, oractivate a drain or a device to pump out the accumulated liquid.

In some embodiments, mounting members of a rack are configured to mountthe servers closely adjacent to one another in the server rack torestrict the flow of the dielectric liquid coolant between thevertically-oriented servers, such that the flow of the dielectric liquidcoolant through the servers is enhanced

In some embodiments, a temperature of a liquid coolant may be monitoredand/or controlled. Methods of monitoring and controlling temperature ofthe oil may be as described in the '579 Publication”), by Best et al.,published Jun. 9, 2011, which is incorporated by reference in itsentirety as if fully set forth herein.

In some embodiments, flow through the servers in augmented usingaugmentation, such as nozzles, fans, or pumps. A separate augmentationdevice may be included on each node, every other node, each row ofnodes, or other frequency. The '579 Publication describes apparatus andmethods using augmentation devices in various embodiments.

In some embodiments, liquid coolant may be removed through the top ofthe rack. Liquid coolant may be reintroduced after having been cooled(for example, by passing the liquid coolant through a heat exchangeroutside of the rack. The '579 Publication describes apparatus andmethods for removing liquid coolant from the top of a rack in variousembodiments.

In various embodiments described herein, computing modules are shown ashaving four racks. Computing modules may nevertheless in variousembodiments have any number of racks. In one embodiment, a computingmodule has one rack.

In various embodiments described herein, a system has been described asholding motherboard assemblies in a submersed or partially submersedcondition. A system may nevertheless in various embodiments hold othertypes of circuit board assemblies or components in a partially submersedcondition.

As used herein, the terms “or” is intended to cover a non-exclusiveinclusion. That is, “or” includes both meanings of both “or” and“and/or.”

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

As used herein, the term “fire-suppressing” means tending to suppress orextinguish, or tending to inhibit propagation of, a fire.Fire-suppressing elements may be active (e.g., a fire suppression systemthat sprays fire suppressant material in response to an alarm), passive(e.g., a pervious concrete material that suppresses fire on a surface,or panel configured to suffocate fire in an enclosed volume), or acombination thereof.

As used herein, the term “data processing module” generally refers toone or more computing devices running software configured to receiverequests, typically over a network. A data processing module may includeone or more servers connected to a network and running softwareconfigured to receive requests from other computing devices on thenetwork, which may include other servers, and desktop and mobilecomputing devices, including cellular phones. Such data processingmodules typically include one or more processors, memory, input/outputconnections to a network and other electronic components, and mayinclude specialized computing devices such as blade servers, networkrouters, data acquisition equipment, disc drive arrays, and otherdevices commonly associated with data centers.

As used herein, the term “node” refers to a computing device that can beconfigured to receive and respond to requests to perform computingoperations. A node may have one processor or multiple processors. Insome embodiments, a node includes one or more servers and/or one or moredata processing modules.

As used herein, the term “tank” refers to a container with or without alid, containing a liquid coolant into which one or more data processingmodules may be installed.

As used herein, an “independently operable” device means capable ofusefully functioning without regard to an operational status of anadjacent device. As used herein, an “independently operable dataprocessing module” means a data processing module that is capable ofusefully functioning to provide data processing services and withoutregard to an operational status of an adjacent data processing module.Operation of independently operable data processing modules can beinfluenced (e.g., heated) by one or more adjacent data processingmodules, but as used herein, an independently operable data processingmodule generally functions regardless of whether an adjacent dataprocessing module operates or is operable.

As used herein, the term “liquid coolant” may be any sufficientlynonconductive liquid such that electrical components (e.g., amotherboard, a memory board, and other electrical or electroniccomponents designed for use in air) continue to reliably function whilesubmerged without significant modification. A suitable liquid coolant isa dielectric liquid coolant, including without limitation vegetable oil,mineral oil, transformer oil, or any liquid coolant have similarfeatures (e.g., a non-flammable, non-toxic liquid with dielectricstrength better than or nearly as comparable as air).

As used herein, “fluid” means either a liquid or a gas, and “coolingfluid” means a gas or liquid coolant typically used for heat-rejectionor cooling purposes. As used herein, a liquid coolant is a subset of theuniverse of cooling fluids, but a cooling fluid may be a dielectric ornon-dielectric liquid or gas, such as, for example, a conventional airconditioning refrigerant.

The flowchart and block diagrams in the drawings illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and program products, according to variousembodiments of the present invention.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what can beclaimed, but rather as descriptions of features specific to particularimplementations of the invention. Certain features that are described inthis specification in the context of separate implementations can alsobe implemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation can also be implemented in multiple implementationsseparately or in any suitable sub combination. Moreover, althoughfeatures can be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination can in some cases be excised from the combination, and theclaimed combination can be directed to a sub combination or variation ofa sub combination.

Similarly, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts.

Further modifications and alternative embodiments of various aspects ofthe invention may be apparent to those skilled in the art in view ofthis description. Accordingly, this description is to be construed asillustrative only and is for the purpose of teaching those skilled inthe art the general manner of carrying out the invention. It is to beunderstood that the forms of the invention shown and described hereinare to be taken as embodiments. Elements and materials may besubstituted for those illustrated and described herein, parts andprocesses may be reversed, and certain features of the invention may beutilized independently, all as would be apparent to one skilled in theart after having the benefit of this description of the invention.Methods may be implemented manually, in software, in hardware, or acombination thereof. The order of any method may be changed, and variouselements may be added, reordered, combined, omitted, modified, etc.Changes may be made in the elements described herein without departingfrom the spirit and scope of the invention as described in the followingclaims.

What is claimed is:
 1. A computing system, comprising: one or more rackrows comprising one or more racks, wherein at least one of the rackscomprises: one or more servers; one or more tanks configured to holdliquid coolant for at least one of the one or more servers; and a liquidcoolant held in at least one of the tanks and configured to remove heatfrom at least one of the one or more servers; an aisle next to one ofthe rack rows or between two of the rack rows, wherein the aislecomprises a floor, wherein the floor can be walked on by servicepersonnel to access at least one of the one or more racks in at leastone of the rows, wherein at least a portion of the floor over at leastone of the cooling components is pervious concrete; and one or morecooling components at least partially below the aisle, wherein at leastone of the cooling components is configured to move a liquid to removeheat from at least one of the servers in at least one of the racks. 2.The computing system of claim 1, wherein the floor and the rackenclosures are at least partially made of fire-retardant materials. 3.The computing system of claim 1, wherein at least a portion of the floorover at least one of the cooling components comprises acoolant-permeable and fire suppressing material.
 4. The computing systemof claim 1, wherein at least one of the one or more cooling componentsconfigured to move the liquid coolant in the tank such that a least aportion of the liquid circulates within the tank, wherein at least aportion of the circulating fluid passes through or across at least oneof the servers to remove heat from the at least one of the servers,wherein the liquid coolant circulates such that substantially all of theliquid coolant on the surface of the liquid coolant contained in the atleast one tank is in motion when the cooling component is operating. 5.The computing system of claim 1, wherein the cooling component comprisesan enclosure, wherein the enclosure is at least partially made of afire-retardant material.
 6. The computing system of claim 1, wherein atleast one of the racks comprises a top cover, wherein the top cover isat least partially made of a fire suppressing material.
 7. The computingsystem of claim 1, wherein the floor forms at least a portion of the topof an enclosure for at least one of the cooling components.
 8. Thecomputing system of claim 1, wherein at least a portion of the floorover at least one of the cooling components comprises one or more floortiles.
 9. The computing system of claim 1, wherein at least one of theracks is mounted on a stand to raise the rack relative to the floor ofthe aisle.
 10. A method of packaging and providing an operatingenvironment for computing resources, comprising: providing one or morerack rows including one or more racks, wherein at least one of the racksincludes one or more servers, the one or more rack rows located on afloor, wherein the floor comprises pervious concrete; at least partiallyfilling at least some of the racks with liquid coolant; and providingone or more cooling components at least partially below the racks,wherein at least one of the cooling components is configured to move aliquid to remove heat from at least one of the servers in at least oneof the racks.
 11. The method of claim 10, wherein providing one or morecooling components comprises enclosing one or more cooling components ina fire-resistant enclosure.
 12. The method of claim 10, furthercomprising circulating a coolant in at least one of the racks such thatfire is suppressed the at least one rack.
 13. The method of claim 10,further comprising collecting coolant on one or more surfaces.
 14. Acooling module, comprising: one or more pumps configurable to move atleast one liquid to remove heat from liquid coolant in at least onerack; and an enclosure, wherein at least one of the pumps is at leastpartially housed in the enclosure, wherein the enclosure comprises atop, wherein at least a portion of the top is coolant-permeable suchthat liquid coolant can pass through the top into the enclosure, whereinat least a portion of the cooling module top is configured to serve as afloor for service personnel standing on the portion to access at leastone of the racks, wherein at least a portion of the top of the enclosurecomprises pervious concrete.
 15. The cooling module of claim 14, whereinthe floor and the enclosure are at least partially made offire-retardant materials.
 16. The cooling module of claim 14, wherein atleast a portion of the top of the enclosure comprises acoolant-permeable and fire-suppressing material.
 17. The cooling moduleof claim 14, wherein at least a portion of the floor top of theenclosure comprises one or more floor tiles.
 18. The cooling module ofclaim 14, wherein the cooling module is configured to at least partiallysupport one or more racks.